Manned, unmanned, and autonomous systems are growing increasingly complex and contain functions with life and safety critical implications. A challenge exists in inadequately testing and evaluating that the implemented systems will reliably meet their design requirements. This activity is sometimes called verification and validation (V&V) or test and evaluation (T&E). Current V&V and T&E techniques are suited for many systems with limited autonomous capabilities, but advanced autonomy and increased complexity makes testing enormously more difficult and in some instances these techniques are no longer suitable.
The V&V and T&E difficulties arise primarily from two characteristics. First, highly autonomous systems have a complex internal state. This can result in an exponential increase in the effort required for V&V or T&E, because the system's internal state becomes part of the space that testing must cover. Consequently, a highly autonomous system interacting with a complex environment represents a challenge for V&V and T&E. Current approaches to achieve high confidence in such systems can be resource, labor, and time prohibitive.
Secondly, most highly autonomous systems are heterogeneous. They are typically constructed in multiple levels, comprising at least low-level control, an intermediate layer of reactive execution, and a high-level mission planning function. In particularly complex cases, there may be more than just three levels. Such heterogeneous systems may pose a special difficulty for testing, because the mapping between different functional layers can become part of the testing problem. It is no longer simply a question of how two or more system components interact. If these components are in different functional layers, the errors or information loss in. the mapping between representations should also be taken into account.
Relevant prior art for V&V approaches can broadly be divided into work on: component V&V for smaller, homogeneous components of high-level control and autonomous systems; constraint-based models for planning, scheduling, and execution; and solving and optimization for hybrid constraint models. Component V&V methods fall broadly into the categories of testing, static checking applied at a source-code level to establish validity of initial values and parameters passed between routines, model-checking methods based either on various kinds of automata or Boolean satisfiability, and automated synthesis methods. Constraint-based models for planning, scheduling, and execution are used to specify and predict behavior, especially for applications of the size and complexity typically found in systems related to space exploration. Solving and optimization of hybrid constraint models is pursued in several separate technical communities. Within Operations Research, Mixed Integer Linear and Nonlinear models are employed. Some researchers in this community have been investigating “mixed logical-linear” methods that integrate mathematical programming with methods for satisfiability or constraint satisfaction more commonly used in Computer Science. The constraint satisfaction community within Computer Science is bridging this gap as well, working in the other direction.
However, current V&V and/or T&E techniques have various limitations such as a limited ability to efficiently and adequately infer system-level properties without exorbitant amounts of testing,